Virtual reality keyboard system and method

ABSTRACT

A system and method implement a virtual reality (VR) keyboard. The VR keyboard system and method receive a VR glove position, generate a corresponding key code from the VR glove position using a predetermined mapping, and send the key code to an application program as a key input corresponding to a keyboard and/or keypad entry of data and/or a command. The system and method also generate a display representing the key input based on the VR glove position. The display of the key input may include, but is to limited to, a displayed depressed key in a VR headset of a VR representation of a VR keyboard indicating the key input. The system and method implementing a virtual reality keyboard addresses and solves numerous difficulties of physical and/or hardware-based input devices and provides many diverse advantages in use and applications.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] This disclosure relates generally to the field of virtualreality, and in particular to a system and method for performingkeyboard inputs without a keyboard using virtual reality.

[0003] 2. Description of Related Art

[0004] The development of graphic user interfaces (GUIs) employing, forexample, “WINDOWS” software, menu-driven software, mouse devices, andtouch screens, has reduced the need for keyboard and keypad entry andfor typing to enter alphanumeric data and commands into a computerand/or other computing-based devices. Such GUIs thus allow users toenter data and commands visually using visually-based windows andscreens, as opposed to tactilely through a keyboard and/or keypad.However, applications using such GUIs are thus beholden to the softwareand software programmers to provide comprehensive windows and menus.Often, a GUI prompts the users to input data and commands through akeyboard and/or keypad by generating an input window or pop-up dataentry line. Accordingly, in some instances, keyboards and/or keypads arenecessities for GUI-based computer interfaces.

[0005] In addition, voice recognition applications have been developedwhich recognize the voice and inflections of speech of users, to allowthe user to enter data and commands orally, as opposed to tactilelythrough a keyboard and/or keypad to enter alphanumeric data andcommands. However, natural language processing may not be 100% accurate,and high accuracy is often gained by providing a great amount ofprocessing power, which may sacrifice the overall performance of thecomputer systems using such voice recognition. Furthermore, to providehigh or perfect accuracy, such voice recognition applications typicallyallow or even require the user to proofread, edit, and correct mistakesthrough alphanumeric entry via a keyboard and/or a keypad. Accordingly,in some instances, keyboards and keypads are necessities forvoice-recognition-based computer interfaces.

[0006] Computer-based devices typically require an input device, anoutput device, and a processor. Heretofore, computers have been madesmaller, portable, handheld, and even wristwatch-sized. However, inorder to provide alphanumeric inputs, keyboards and/or keypads havegenerally been requisite components. The reduction of size of suchkeyboards has been difficult, since a wide range of alphanumeric keys istypically required. For example, keyboards based on the Roman and/orEnglish alphabet require at least 26 keys, and enhanced keyboardsrequire over 90 keys for number, cursor arrows, a space bar, punctuationsymbols, controls such as TAB and ESC, etc. To accommodate such keys,manufacturers may reduce the sizes of individual keys, which sacrificesease of use by the user. In addition, keyboards/keypads have heretoforebeen primarily hardware-based, and so are susceptible to wear, damage,and theft.

[0007] One proposed solution to provide reduced size and/or portabilityincludes the use of telephone-based keypads having about 10 or morekeys, with some keys having letters thereupon, such as the 2 key havingthe letters A, B, and C, thus resembling a telephone keypad such as adual tone multiple frequency (DTMF) telephone. Through judicioushardware and/or software implementations, a user may be able to enterentire words and sentences, with sufficient patience and practice.

[0008] Other proposed solutions include eliminating the keyboard/keypadand instead using a touchscreen with handwriting recognition, such asthe alphanumeric entry system of the “PILOT” handheld device. However,handwriting recognition systems typically require training the user toinput a set of predefined symbols representing alphanumeric charactersinstead of the actual alphanumeric characters. Accordingly, keyboardsand the like have not be replaced with equivalent or better devices.

[0009] A need exists for a system and method for providing theversatility of a keyboard and/or keypad for alphanumeric entry withoutrequiring a keyboard and/or keypad and without sacrificing theadvantages of a keyboard and/or keypad for inputting a large range ofdata and/or commands.

[0010] Virtual reality (VR) applications have heretofore provided arudimentary system, typically using a GUI, for manipulatingrepresentations of physical elements, such as virtual reality doors andelevators in virtual worlds. Typically implemented in software, such VRapplications interface with headsets and gloves to respond to and tointerpret the movements of the head and hands of the user as commandsand control signals. Heretofore, such VR applications and VR worlds areGUI-based, and so do not provide the versatility of hardware-based,actual and physical keyboards and/or keypads.

SUMMARY OF THE INVENTION

[0011] It is recognized herein that a system and method for providing avirtual reality keyboard addresses solves numerous difficulties of inputdevices and provides many diverse advantages in use and applications.

[0012] A system and method are disclosed which receive a virtual reality(VR) glove position, which generate a corresponding key code from the VRglove position using a predetermined mapping, and which send the keycode to an application program as a key input corresponding to akeyboard and/or keypad entry of data and/or a command. The system andmethod also generate a display representing the key input based on theVR glove position. The display of the key input may include, but is notlimited to, a displayed in a VR headset of a VR representation of a VRkeyboard indicating the key input.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The features and advantages of the disclosed virtual realitykeyboard system and method are readily apparent and are to be understoodby referring to the following detailed description of the preferredembodiments of the present invention, taken in conjunction with theaccompanying drawings, in which:

[0014]FIG. 1 illustrates a schematic of a first embodiment of thedisclosed VR keyboard system and method;

[0015]FIG. 2 illustrates the VR keyboard system and method of FIG. 1 inuse by a user;

[0016]FIG. 3 illustrates a mapping of the VR glove positions to keycodesand displayed keys;

[0017]FIG. 4 illustrates a flowchart of the method of operation of theVR keyboard system and method of FIG. 1;

[0018]FIG. 5 illustrates the VR keyboard system and method having a VRworld displaying the VR keyboard;

[0019]FIG. 6 illustrates a flowchart for operating the VR keyboard withthe VR world of FIG. 5;

[0020]FIG. 7 illustrates a second embodiment of the VR keyboard systemand method using different keyboard mappings to provide different VRkeyboards;

[0021]FIG. 8 illustrates a schematic of the second embodiment of FIG. 7for implementing different VR keyboards;

[0022]FIG. 9 illustrates a mapping used by the second embodiment toimplement different VR keyboards with different character sets andlayouts;

[0023]FIG. 10 illustrates a VR keyboard with an ergonomic layout;

[0024]FIG. 11 illustrates a VR keyboard having special function keys;

[0025]FIG. 12 illustrates a VR keyboard for the Greek alphabet;

[0026]FIG. 13 illustrates a VR keyboard for the Cyrillic alphabet;

[0027]FIG. 14 illustrates a VR keyboard with word processing commandkeys;

[0028]FIG. 15 illustrates a VR keyboard for a typesetting application;

[0029]FIG. 16 illustrates a VR keyboard with specialized keys for acomputer game;

[0030]FIG. 17 illustrates a VR keyboard for providing a piano-likekeyboard;

[0031]FIG. 18 illustrates a third embodiment of the VR keyboard systemand method using a force feedback VR glove;

[0032]FIG. 19 illustrates a flowchart of a method implementing the thirdembodiment of FIG. 18;

[0033]FIG. 20 illustrates a schematic of a fourth embodiment of the VRkeyboard system and method using a neural network;

[0034]FIG. 21 illustrates a flowchart of a method for training theneural network in the fourth embodiment of FIG. 20;

[0035]FIG. 22 illustrates a flowchart of a method using the VR keyboardsystem of FIG. 20 having a trained neural network;

[0036]FIG. 23 illustrates a flowchart of an alternative method fortraining the neural network in the fourth embodiment of FIG. 20 toprovide user authentication;

[0037]FIG. 24 illustrates a flowchart of a method for authenticating auser to use the VR keyboard system of FIG. 20;

[0038]FIG. 25 illustrates a fifth embodiment of the VR keyboard systemand method using an auto-hide feature to hide the VR keyboard;

[0039]FIG. 26 illustrates the fifth embodiment of FIG. 25 displaying theVR keyboard when a user has the headset oriented in a predeterminedorientation;

[0040]FIG. 27 illustrates a flowchart of a method of operation using theauto-hide feature in the fifth embodiment of the disclosed VR keyboardsystem and method;

[0041]FIG. 28 illustrates a flowchart of a method for determining arange of orientations for use in the fifth embodiment;

[0042]FIG. 29 illustrates a flowchart of a method for toggling theauto-hide feature of the fifth embodiment;

[0043]FIG. 30 illustrates a sixth embodiment of the disclosed VRkeyboard system and method using both a VR keyboard and an actualphysical keyboard;

[0044]FIG. 31 illustrates a schematic of the sixth embodiment of FIG.30;

[0045]FIG. 32 illustrates a flowchart of a method for sampling controlsignals from an actual keyboard for use by the VR keyboard;

[0046]FIG. 33 illustrates a flowchart of a method using sampled controlsignals to operate the VR keyboard;

[0047]FIG. 34 illustrates a mapping of glove positions with displayedkeys and sampled control signals;

[0048]FIG. 35 illustrates a seventh embodiment of the VR keyboard systemand method with a VR mouse;

[0049]FIG. 36 illustrates the seventh embodiment using the VR glove tooperate the VR mouse;

[0050]FIG. 37 illustrates VR glove positions for keyboard use;

[0051]FIG. 38 illustrates VR glove positions for mouse use;

[0052]FIG. 39 illustrates a flowchart of a method for operating theseventh embodiment using either a VR keyboard or a VR mouse;

[0053]FIG. 40 illustrates an eighth embodiment of a VR keyboard and VRmouse for use with an actual keyboard and actual mouse;

[0054]FIG. 41 illustrates a flowchart of operation of the eighthembodiment to respond to either the VR keyboard or the actual keyboard;

[0055]FIG. 42 illustrates a flowchart of operation of the eighthembodiment to respond to either the VR mouse or the actual mouse;

[0056]FIG. 43 illustrates a ninth embodiment of the disclosed VRkeyboard system and method for displaying VR hand images using the VRkeyboard;

[0057]FIG. 44 illustrates a flowchart of the method of operation of theninth embodiment for displaying a VR hand image;

[0058]FIG. 45 illustrates a flowchart of a method for generating the VRhand images of the ninth embodiment;

[0059]FIG. 46 illustrates a tenth embodiment of the disclosed VRkeyboard system and method using sensors to detect hand positions of auser without a VR glove;

[0060]FIG. 47 illustrates an alternative of the tenth embodiment using acamera and machine vision to detect hand-positions of a user without aVR glove;

[0061]FIG. 48 illustrates another alternative of the tenth embodimentusing an infrared detector to detect hand positions of a user without aVR glove;

[0062]FIG. 49 illustrates a schematic of the tenth embodiment using handsensors for operating a VR keyboard; and

[0063]FIG. 50 illustrates a flowchart of the operation of the tenthembodiment for scanning hand positions of the hands of the user tooperate the VR keyboard.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0064] Referring in specific detail to the drawings, with commonreference numbers identifying similar or identical elements, steps, andfeatures, as shown in FIG. 1, the present disclosure describes a system10 and method for generating and operating a virtual reality (VR)keyboard 12. The term “keyboard” is defined herein to includealphanumeric keyboards, subsets of alphanumeric keyboards, keypadsincluding numerical keypads, telephone and DTMF keypads, security accessinput devices using buttons with labels, etc., and so is not limited toQWERTY alphanumeric keyboards. Accordingly, it is understood that theuse of the term “keyboard” and the depiction in any figures of akeyboard such as a QWERTY alphanumeric keyboard typically used withpersonal computers and the like is only an example of a keyboard foruse, interaction, and operation by a user for any application ofkeyboards for input and/or output devices. As defined herein, the term“keyboard” is more than a plurality of keys, since a keyboard includes alayout of the plurality of keys as well as the keys, with the layouttypically being predetermined. The keys may be associated with symbolssuch as alphabetical, numerical, mathematical, or other representations,and the keys may include associated pictorial or symbolicrepresentations thereupon. Accordingly, a keyboard is not identical to aset of buttons, but may be a plurality of buttons having a layout and aset of symbols associated with each key or button.

[0065] The term “virtual reality” and its abbreviation “VR” are hereindefined to include, but not to be limited to, visual and/or othersensory applications implemented using software and/or hardware tosimulate and/or provide representations of environments which may bedifferent from the physical environment of the user. Such VR may providevisual and/or multimedia zones, worlds, and work areas in which the userand/or other software applications may change and interactrepresentations of elements in the VR environment. For example, in a VRworld, a graphic representation of a switch may be changed to representthe flicking or switching of the switch, which may have an associatedswitch-flicking sound which is activated by flicking the switch. Inaddition, the VR switching of the VR switch may cause the actuation ofother events, either in the VR world or in actual physical devices andstructures; for example, the flicking of the VR switch may cause anactual computer to be turned on or off. Accordingly, the term “virtualreality” is not limited to simulations or representations of VR devicesand information in VR worlds, but may also be extended to physicaldevices as well as, in hybrid implementations, to both physical and VRdevices.

[0066]FIG. 1 illustrates a schematic of a first embodiment of thedisclosed VR keyboard system and method, with the VR keyboard systemincluding a VR headset known in the art to be worn or attached to auser, such that at least one display is visible to the user. Forexample, the display may be a liquid crystal display (LCD), known in theart, for generating images with depth and/or perspective to display a VRrepresentation of a keyboard 12, as shown in FIG. 2, as well as VRworlds and environments. The LCD may be visible to one or both eyesand/or may be bifocal; that is, the user may view the VR world and/orkeyboard by titling an eye in one direction toward the LCD, and may viewthe actual physical environment through the LCD by tilting an eye inanother direction. The VR headset and/or LCD receive signals from afirst processor through a first channel, which may be a wire, a wirelessconnection, a fiber optic, etc. The first processor is a VR processorwhich executes a VR program generated from source code in programminglanguages such as C++, “VISUAL BASIC”, or Virtual Reality Mark-UpLanguage (VRML) to send VR images to the LCD and thence for viewing bythe user. In addition or alternatively, the VR program may be at leastone object in an object oriented programming language, which may beevent-driven and/or which may poll input devices such as the VR glove toreceive and process data to operate the VR keyboard.

[0067] The first processor may include a microprocessor such as a“PENTIUM” microprocessor with memory, such as about 16 MB of RAM forexecuting the VR program. The first processor may be a desktop personalcomputer (PC), a workstation, or a portable or laptop computer. Forexample, the first processor may be worn or mounted on the user; forexample, on a belt about the waist of the user as the user wears the VRheadset.

[0068] The disclosed VR keyboard system also includes at least one VRglove. In a preferred embodiment, two VR gloves are used to provide theuser with full multi-hand functionality for VR keyboard inputs. However,one VR glove may be used for specific applications, such as numericaldata entry and/or telephone number entry using a VR numerical keypad.

[0069] The VR glove is connected to the first processor through a secondchannel, which may be a wire, a wireless connection, a fiber optic, etc.for providing glove position signals to the first processor. The VRglove includes sensors for detecting the position and/or orientation ofportions of the hand or hands of the user. As defined herein the term“position” of the VR glove refers to the direction and orientation ofportions of the VR glove as well as a position, relative or absolute, ofthe VR glove and portions thereof as the VR glove is manipulated by theuser, and also the configuration of the VR glove, including the aspects,bending, and lengths of the fingers of the user.

[0070] Some people do not consider the thumb to be included in the term“finger”. However, as defined herein, the term “finger” with referenceto the user and/or the VR glove includes the thumb; the index, middle,and ring fingers; the little or pinky finger; any digits of the hand andportions thereof; any or all toes; and other portions of the body,including portions of the body such as discreet and/or relativelyobscured body parts not otherwise specified herein. One may chose toimplement the disclosed VR keyboard system and method using musclemovements not limited to hand movements to enter alphanumeric data andcommands through the VR keyboard. Accordingly, the term “finger” is notlimited to any particular portion of the hand of the user.

[0071] The first processor generates key input signals such as key codescorresponding to the position of the VR glove to represent motions ofthe user to input alphanumeric data and/or commands through the VRkeyboard. As defined herein, the term “alphanumeric” includes, but isnot limited to, predetermined alphabets such as the English, Romanand/or Greek alphabets, predetermined number systems such as the Arabicnumber system, pictorial and/or pictographic languages such as theJapanese and Chinese languages, typographical systems, mathematical andchemical symbols, etc., and so is not limited to the symbols on a QWERTYkeyboard. The key input signals are output to a second processor for usewith application programs, including input and/or keyboard driversoftware, which reside in memory and which are executed by the secondprocessor, for example, to receive and process input signalscorresponding to the actuation of keys of a physical keyboard. Inaddition, the first processor may be incorporated within or be acomponent of the second processor.

[0072] In operating the application program, the second processor maydisplay screens to implement, for example, graphic user interfaces(GUIs), as well as to receive and transmit other signals, for example,through other input and/or output devices, such as a mouse input, anactual keyboard, scanners, printers, modems, connections to networkssuch as the Internet, etc. For example, the application program may bethe “ACCESS” database, the “EXCEL” spreadsheet, the “WORD”wordprocessor, the “INTERNET EXPLORER” Internet browser, the “MONEY”financial program, etc., all of such application programs beingavailable from “MICROSOFT CORPORATION”. Such application programs arecapable of receiving inputs corresponding to keyboard inputs, andgenerate outputs which may be displayed on a display, including acomputer monitor and/or the LCD of a VR headset. Accordingly, displaysignals generated by the second processor may be transferred through thefirst processor to the LCD of the VR headset.

[0073]FIG. 2 illustrates the VR keyboard system and method of FIG. 1 inuse by a user, with the user operating the VR gloves, connected to thefirst processor, to manipulate the VR keyboard in the view of the user.The display may be an actual display device connected to the secondprocessor and/or may be a VR representation of a display screen. The VRkeyboard may be superimposed in the view of the user to appear in theregion of space near or in front of the display, so that the user hasthe perception of viewing the VR keyboard as an actual keyboard foroperation with the display displaying screens such as a GUI foroperating the application program. In addition, the VR keyboard systemand method may generate the VR display to also be superimposed over theactual display, so that the VR display may complement the actual displayand the operation of the application program displayed on the actualdisplay. The VR display may display the application program or otherprograms operating concurrently, for example, secondary windows of a GUIrelative to primary windows displayed by the actual display.

[0074]FIG. 3 illustrates a mapping of the VR glove positions to keycodesand displayed keys, with the VR glove positions, such as vector, radial,or cylindrical coordinates, being mapped to a predetermined mapping ofkey positions or coordinates, which are in turn mapped to keycodes,which may be ASCII codes or signals for use by a device for generating aspecific keystroke of a keyboard. The key codes are then mapped tocorrespond to a displayed key and/or symbol. For example, the VR gloveposition or coordinates (X3, Y3, Z3) correspond to the positions orcoordinates (XA, YA, ZA) in VR on the VR keyboard to provide depthand/or perspective in the screen representation of the VR world. Indisplaying the VR world and the VR keyboard, the first processor mayconvert the VR coordinates (XA, YA, ZA) to corresponding two dimensionalscreen coordinates for display while providing associated depth and/orperspective.

[0075] The VR coordinates (XA, YA, ZA) in turn correspond, for example,to an ASCII value 65 which is used to display a capital letter “A”, inthe Roman alphabet. As shown in FIG. 2, other glove positions maycorrespond to other letters such as lower case “p”, as well as controlcodes such as an ASCII space labelled (SPACE) and other control signalssuch as the Escape (ESC) key, the Control (CTRL) key, number keys, theScroll Lock key., etc. on computer keyboards.

[0076]FIG. 4 illustrates a flowchart of the method of operation of theVR keyboard system of FIG. 1. The VR keyboard system and method maydisplay a VR representation of a keyboard as the VR keyboard to the userthrough the VR headset, and the user may then move the VR glove to pressthe VR keys on the VR keyboard. The VR keyboard system and method get aVR glove position corresponding to the motions of the user to press theVR keys. The VR glove position may be obtained either passively byreceiving VR glove position data, including using event-driven dataacquisition software, or actively by polling the VR glove for positiondata at intervals, which may be periodic, regular, or irregular. The VRkeyboard system and method then generates a corresponding key code fromthe VR glove position using a predetermined mapping, such as shown, forexample, in FIG. 3.

[0077] The method has the first processor send the key code as acorresponding key input signal to the application program and/or to theinput or keyboard driver software; for example, in the same manner asthough an actual keyboard had generated and sent a key input signal tothe application program or keyboard driver software in response toactuation of a physical key by the user. The application programprocesses the key input signal to perform appropriate and/orcorresponding tasks, such as receiving input numerical data andperforming number crunching data display.

[0078] The method then generates a display representing the key input;for example, a VR display generated in the VR headset may be changed torepresent a VR representation of the key input such as a simulation of acorresponding key being depressed, with such a VR representation beingsent to the VR headset for display to the user. Accordingly, as the userinputs data by depressing VR keys in the VR world having a VR keyboardbeing displayed to the user, corresponding key inputs are applied to theapplication program, and the actions of the user are displayed to theuser as VR representations of the user pressing the VR keys.

[0079] The method may then loop back to get additional VR glovepositions. In addition or alternatively, the method may perform thesteps of FIG. 2 concurrently, for example, in a pipelined and/orparallel processing manner to respond to VR glove positions to actuateVR keys and to update the VR representation of the VR keyboard.

[0080] Accordingly, the user may operate and interact with theapplication program without the need for an actual physical keyboard.Since the VR keyboard may be implemented in software using the VRprogram, and the VR headset and VR gloves are employed, the VR programmay be configured to provide VR keyboards of any size, any layout, andany character set, and so are not limited to a fixed physical and oftenunalterable keyboard. In addition, the VR keyboard may be utilizedrepeatedly without the concerns of wear on physical components. The VRprogram may be readily copied and backed up, within the legallimitations of copyright and other applicable rights. The varioushardware components of the VR keyboard system 10 may be replaced and/orrepaired as necessary to address wear and tear. In addition, such VRgloves and VR headsets may be relatively compact compared to the typicalphysical keyboard spanning about 18 inches (45 cm.) across. Accordingly,the disclosed VR keyboard system and method are applicable to laptop andhandheld computing devices. In addition, although the VR keyboard systemand method may operate using the VR headset in conjunction with adisplay associated with the second processor, the display of the VRheadset may be used instead of the separate display or monitor.

[0081] In addition, devices referred to as network computers (NCs)purport to provide computing capabilities without the need for largeamounts or even any hard drive and/or local memory. Instead, such NCsprovide a display, an input device, and an interface to a network whichremotely operates application programs in remote memory. Using the VRkeyboard system and method, the first processor may act as an NC withouta physical keyboard and optionally without the physical display separatefrom the LCD of the VR headset. Accordingly, the first processor mayinclude network interface software and/or hardware to connect the VRkeyboard system and method to a network, such as the Internet and theWorld Wide Web, to operate application programs remotely using the VRkeyboard. In addition, such remote applications of the disclosed VRkeyboard system and method may be used to control appliances and devicesremotely, in situations in which such appliances and devices requiresome alphanumeric input. For example, one may set a programmable videocassette recorder (VCR) or the thermostat of a house using applicationsoftware adapted to respond to such alphanumeric inputs providedremotely from the disclosed VR keyboard system and method.

[0082] Furthermore, since the VR keyboard system and method relies onrelative VR glove positions, orientations, and configurations, the VRkeyboard system and method may be used with the user oriented in anyposition, including lying down, at any angular position relative to anyframe of reference, and even upside down. In addition, the use of the VRkeyboard may be performed by the user moving the fingers and handsthereof in any comfortable manner. Accordingly, carpal tunnel syndromemay be reduced or even eliminated, since the user is not required torest the wrists on a physical surface or to orient the arms in any fixedorientation in order to actuate the VR keyboard.

[0083] Moreover, the VR keyboard system and method is notgravity-dependent or pressure-dependent to provide a VR keyboard to theuser. Accordingly, the disclosed VR keyboard system and method may beused in low-gravity or zero-gravity environments such as in outer spacein space stations, space vehicles, lunar or Mars landing craft or bases,etc., as well as underwater, in bathyspheres, in air balloons such ashigh altitude balloons, and other environments with high or low ambientpressure such as air pressure and water pressure.

[0084]FIG. 5 illustrates the VR keyboard system and method having a VRworld displaying the VR keyboard. The VR world may be displayed on theLCD of the VR headset to provide a VR office setting and/or a VR desktopwith a VR representation of a computer monitor/display and a VR mouse ona VR desk. The VR world may also include a VR bookcase for accessinginformation indexed through VR books, which may actuate databasesassociated with the application programs and/or the second processor. AVR stereo system may be provided for accessing a VR tuner and/or VRmedia players to activate actual media players implemented in hardwareand/or software associated with the application programs and/or thesecond processor.

[0085]FIG. 6 illustrates a flowchart for operating the VR keyboard withthe VR world of FIG. 5, in which the step of generating the displayrepresenting a key input includes the steps of generating the VR worldrepresentations, and generating the key input display on the VRrepresentation of the VR keyboard in the VR world representation in theLCD of the VR headset.

[0086]FIG. 7 illustrates a second embodiment of the VR keyboard systemand method using different keyboard mappings to provide is different VRkeyboards, such as an ergonomic or butterfly-shaped keyboard, whichstores in memory and uses at least one predetermined keyboard mapping.

[0087]FIG. 8 illustrates a schematic of the second embodiment of FIG. 7for implementing different VR keyboards, in which a first predeterminedkeyboard mapping may include a keyboard character set such as for anenhanced keyboard, a numerical keypad, a DTMF keypad, a Dvorak keyboard,a Greek or Cyrillic alphabet character set, customized anduser-customizable character sets, Chinese and other pictographiclanguage character sets, and other sets of symbols such as mathematicalsymbols.

[0088] The first predetermined keyboard mapping may include VR keyboardlayouts, such as data specifying a three dimensional keyboard, a twodimensional keyboard, an ergonomic keyboard layout, and even musicalinstrument key layouts for implementing VR representations of a piano, amusic synthesizer, an organ, etc. Additional predetermined keyboardmappings may also be stored for implementing VR keyboards withcommand/control codes to control game command codes, word processorcodes and hot keys, programming language codes such as assembly languagemnemonics, etc. The second predetermined keyboard mapping may alsoinclude a command/control layout for configuring the positioning of suchcommand/control code keys.

[0089]FIG. 9 illustrates a mapping used by the second embodiment toimplement different VR keyboards with different character sets andlayouts, in which the VR glove positions are mapped to a predeterminedmapping, which then maps to corresponding keyboards and layout data. Thekeycodes are used to determine the displayed keys actuated by the userthrough the V R glove, and the layout data specifies where, within theVR representation of the VR keyboard and in the LCD of the VR headset,to display the actuated VR keys. For example, the glove position (X3,Y3, Z3) maps to a VR glove position. (XA, YA, ZA) for a capital “A”. Thekeycode 65 is then generated to display an “A”, and the layout data(XLA, YLA, ZLA) determines how and where in the VR representation thedisplayed “A” is to be shown. The layout data may also include otherfeatures, such as perspective, shading, color changes, etc., to indicateto the user that the VR key has been actuated or “depressed” by the userthrough the VR glove interacting with the VR keyboard.

[0090]FIG. 10 illustrates a VR keyboard with an ergonomic layout, whichmay provide a more relaxed-orientation for use by the user to input dataand/or commands. FIG. 11 illustrates a VR keyboard including additionalspecial function keys, such as a “WINDOWS START BUTTON” key, a “WINDOWS”menu pointer key, etc., such as those keys found on 105+ keyboards foruse with “WINDOWS 95” available from “MICROSOFT CORPORATION”. Otherspecial function keys may include TRANSMIT and KWIC used on keyboardinterfaces for accessing the “WESTLAW” legal database.

[0091]FIG. 12 illustrates a VR keyboard for the Greek alphabet. FIG. 13illustrates a VR keyboard for the Cyrillic alphabet. FIG. 14 illustratesa VR keyboard with word processing command keys, such as SPELL, SCREEN,SEARCH; etc. used in “WORDPERFECT” available from “COREL”. FIG. 15illustrates a VR keyboard for a typesetting application, with an ETAOINSHRDLU key arrangement.

[0092] Other formats may also be included such as the Dvorak keyboard.Since some consider the QWERTY keyboard to be an accident of historywhich has caused technological lock-in to an allegedly inefficientkeyboard layout, the disclosed VR keyboard system and method mayimplement the Dvorak keyboard and any other formats which may be moreefficient. Accordingly, the disclosed VR keyboard system and methodfrees typists of such alleged technological lock-in. In addition, thekeyboard may be customized using a customize keyboard layout subroutineimplemented by the first processor. For example, left-handed people maycompletely reverse the layout of the standard QWERTY keyboard to form amirror image adapted for left-handed people; for example, to have theright hand positioned to control, for example, the keys for ASDF, and somay form what may be called a YTREWQ keyboard, or alternatively a POIUYTkeyboard.

[0093]FIG. 16 illustrates a VR keyboard with specialized keys for acomputer game, such as a “STAR TREK” game available through “PARAMOUNT”for providing commands to activate phasers, photo torpedoes, scanningfunctions, starship acceleration and deceleration, etc. corresponding toactuatable commands in the game. Other computer games such as “DOOM”,“HERETIC”, and “ULTIMA III” use the keyboard to implement game commands.The disclosed VR keyboard system and method may be used to implementsuch layouts. In addition, such game command layouts may be loaded intothe VR program as configuration files during or after installation ofthe computer game.

[0094]FIG. 17 illustrates a VR keyboard for providing a piano-likekeyboard, which allows the user to actuate the VR keys to perform actualmusic through the second processor and multimedia application programs.

[0095]FIG. 18 illustrates a third embodiment of the VR keyboard systemand method using a force feedback VR glove, in which the positionsignals from the VR glove are provided to the first processor, which inturn generates and sends actuation signals to a plurality of actuators,for example, in the tips of the fingers of the VR glove. FIG. 19illustrates a flowchart of a method implementing the third embodiment ofFIG. 18, in which the VR keyboard system gets the VR glove position, anddetects for motions of the fingers corresponding to the engaging anddepressing of the VR keys. Subsequent to or concurrent with thegeneration of key codes and the other steps of FIG. 4 to perform the VRkeyboard method, the disclosed VR keyboard system and method generatesactuation signals corresponding to the keys depression motions, andapplies the actuation signals to the actuators in the VR glove toprovide force feedback to the user. Accordingly, the user is providedwith the physical sensation of depressing an actual keyboard key, wheninstead the user has depressed in VR a VR key of the VR keyboard.

[0096]FIG. 20 illustrates a schematic of a fourth embodiment of the VRkeyboard system and method using a neural network, which is implemented,for example, in the first processor. In particular, the neural networkmay be characterized as parameters of a plurality of nodes stored inconjunction with the predetermined keyboard mapping in the memory of thefirst processor. In addition or alternatively, the neural network may beimplemented in an integrated circuit.

[0097]FIG. 21 illustrates a flowchart of a method for training theneural network in the fourth embodiment of FIG. 20, in which the VRkeyboard system and method displays a set of predetermined VR keys tothe user through the LCD of the headset, or alternatively through thedisplay associated with the second processor. The display of keys isprovided for user interaction to prompt the user to activate the VR keysas specified VR keys are displayed. The set of predetermined keys mayinclude at least one pass through every available VR key on the VRkeyboard.

[0098] The disclosed VR keyboard system and method then get VR glovepositions corresponding to the displayed predetermined VR keys depressedin VR by the user, and generate corresponding key codes from the VRglove positions using the predetermined mapping. The key codes and theVR glove positions are sent as inputs to the neural network fortraining, such that the trained neural network recognizes the VR glovemotions as corresponding to VR key actuations. The trained neuralnetwork may be stored as a predetermined neural network mapping in thememory of the first processor.

[0099] The trained neural network may thus be used to allow the user toperform touch typing without viewing the VR keyboard. Accordingly, theVR keyboard may be displayed during the training period, and,optionally, to not be displayed during use by the user to input data andcommands. The user may then touch type using an “invisible” keyboard;that is, the keyboard is not physical, yet not displayed to the user.Such VR touch typing using an invisible VR keyboard may be used toreduce the computational power required to operate the VR program, sincethe step of updating the VR keyboard to reflect VR actuation of the VRkeys is eliminated.

[0100]FIG. 22 illustrates a flowchart of a method using the VR keyboardsystem of FIG. 20 using a trained neural network, which gets the VRglove positions and then applies the VR glove positions to the trainedneural network. The neural network then generates a prediction of the VRkey actuated or intended to be actuated by the user from the VR glovepositions, and in turn generates the key code from the VR key actuationprediction using the predetermined neural network mapping.

[0101]FIG. 23 illustrates a flowchart of an alternative method fortraining the neural network in the fourth embodiment of FIG. 20 toprovide user authentication. The disclosed VR keyboard system and methoddisplays a predetermined set of VR keys and/or a predetermined VRkeyboard layout for interaction with a specified user. The user may bespecified by a name, an identification number, a password, a biometriccharacteristic, etc.

[0102] The user is prompted to position the VR glove such that the usersees a VR glove representation, generated by the disclosed VR keyboardsystem and method, to be substantially adjacent to the VR keyboarddisplayed in the LCD of the headset. The user is prompted to interact inVR using the VR glove representation to depresses VR keys by, forexample, typing in VR a predetermined sequence of VR keys, or a randomlygenerated sequence of VR keys. Alternatively, the user may be promptedto position the VR glove on an active or inactive physical keyboard toprovide cues to the user to guide the positioning of the VR glove andfingers.

[0103] The disclosed VR keyboard system and method then get the VR glovepositions corresponding to the VR interactions of the VR glove with theVR keyboard or the actual keyboard, and send the VR glove positions tothe neural network for training to recognized the specific user by thepositioning and use of the VR gloves on the VR keyboard or on the actualkeyboard.

[0104] Such positioning and use of the VR gloves corresponds to how thespecific user interacts with a keyboard, whether an actual keyboard or aVR keyboard, and such positioning and use may be sufficiently unique tothe specific user in the positioning, orientation, and aspects of thehands, in the mannerisms of the user during typing, in the speed andreaction response times, etc. Accordingly, hand positioning duringtyping may function as a biometric uniquely associated with the user.

[0105]FIG. 24 illustrates a flowchart of a method for authenticating auser under test to use the VR keyboard system and method of FIG. 20.Once the neural network is trained, the disclosed VR keyboard system andmethod displays the same predetermined sequence of letters, the samerandomly generated sequence of keys, or even a new random set of keysfor the user under test to actuate in VR. When the user under test isinteracting to enter the displayed keys, the VR keyboard system andmethod get the VR glove positions, and apply the VR glove positions andthe key codes of the displayed keys to the neural network. The neuralnetwork then classifies the VR glove positions as corresponding to ornot corresponding to the specific user within a predetermined errortolerance; for example, 95% accuracy.

[0106] The VR keyboard system and method then generate a control signalin response to the classification, and provides access or denial ofaccess of the user under test to an application program through the VRkeyboard in response to the control signal.

[0107]FIG. 25 illustrates a fifth embodiment of the VR keyboard systemand method using an auto-hide feature to hide the VR keyboard, in whichthe user, with the headset tilted in a first direction, does not see aVR keyboard. FIG. 26 illustrates the fifth embodiment of FIG. 25displaying the VR keyboard when a user orients the headset in apredetermined orientation, for example, at an angle α below thehorizontal. Since the VR keyboard system and method may be used in anyorientation, a reference orientation may be, for example, a surfaceperpendicular to a flat portion of the LCD of the headset.

[0108]FIG. 27 illustrates a flowchart of a method of operation using theauto-hide feature in the fifth embodiment of the disclosed VR keyboardsystem which gets the headset orientation from orientation sensors onthe headset. The disclosed VR keyboard system and method then determineif the headset orientation is within a predetermined range; for example,greater than 30° below the horizontal. If so, a VR keyboard displaysignal is generated.

[0109] Concurrent with getting VR glove positions for VR keyboardprocessing, the method responds to the VR keyboard display signal todisplay or not to display the VR keyboard in the LCD of the VR headset.If the VR keyboard display signal indicates displaying the VR keyboard,the method generates the VR keyboard representation concurrent with theprocessing of the VR glove positions for receiving, processing, andindicating in VR the VR keyboard input. If the VR keyboard displaysignal indicates to not display the VR keyboard, the method does notgenerate the VR keyboard representation but continues processing of theVR glove positions for receiving and processing the VR keyboard input.

[0110] The method thus provides an auto-hide feature, in which the VRkeyboard is hidden when the user is focusing on the actual display, oralternatively is focusing on the LCD display with the head tilting in acertain range of orientations. In this manner, user may be intent onviewing a portion of the display; for example, the application programsuch as a spreadsheet, and does not require looking at a keyboard,whether an actual keyboard or a VR keyboard. Alternatively, the user maybe touch typing in VR, and so may be distracted by the view of the VRkeyboard.

[0111] If the user chooses to focus on the keyboard; for example, if theuser makes a typing mistake and/or cannot touch type, the user looksdownward as though an actual keyboard is in front of the user.Accordingly, the disclosed VR keyboard system and method provide theuser with the traditional environment of typing with a physicalkeyboard, including looking down and focusing on a keyboard, without theuse of an actual keyboard. In addition, since the VR keyboard is notalways displayed, the use of the auto-hide feature may reduce thecomputational power required to display and update the VR keyboard.

[0112] The range of orientations of the auto-hide feature may bepredetermined, for example, head tilts of greater than 30° may be usedas default values, or the user may specify a different value.Alternatively, FIG. 28 illustrates a flowchart of a method fordetermining a range of orientations for use in the fifth embodiment. TheVR keyboard may be set to be viewed at a predetermined angle below ahorizontal line or surface, or the angle may be set randomly. As aspecific user uses the VR keyboard, the method gets the headsetorientations during use, and learns the range of headset orientationcorresponding to the specific user looking at the VR keyboard, forexample, to determine a relative comfort level for the specific user.

[0113] The learning step may be performed by a neural network, by aprocessor determined an average value of the angular tilting by the userover a duration of use, or by other techniques for optimizing a range ofangles associated with the specific user. For example, the learning stepmay include detecting for the VR keyboard being displayed in apredetermined percentage of the overall VR display; that is, the usermay require viewing over 80% of the VR keyboard to effectively operatethe VR keyboard. The learning step may then average the percentages overa duration of use, and determine an optimum range of angles forproviding such an average percentage. The learned range is then storedin memory as the predetermined range.

[0114]FIG. 29 illustrates a flowchart of a method for toggling theauto-hide feature of the fifth embodiment. The auto-hide feature may bean optional feature which may be toggled by a predetermined VR auto-hidetoggle command. Upon detecting such a command, if auto-hide is set to bedisabled, the method disables the auto-hide feature and always displaysthe VR keyboard. Otherwise, if the auto-hide feature is set to beactive; that is, to auto-hide the VR keyboard, then auto-hide isperformed such that the VR keyboard is generated and displayed only ifthe user is looking down within a predetermined range. The method maythen loop back and continually check for the toggle command, using, forexample, an event-driven object of object oriented programmingresponding to the toggling of the auto-hide feature.

[0115]FIG. 30 illustrates a sixth embodiment of the disclosed VRkeyboard system and method using both a VR keyboard and an actualphysical keyboard, in which the actual keyboard may be used to generatethe predetermined mapping to be used by the VR keyboard. FIG. 31illustrates a schematic of the sixth embodiment of FIG. 30, in which theactual physical keyboard is connected to the second processor, oralternatively to the first processor, through a third channel, which maybe a wire, a wireless connection, a fiber optic, etc.

[0116]FIG. 32 illustrates a flowchart of a method for sampling controlsignals from an actual keyboard for use by the VR keyboard, in which theVR keyboard system and method displays through the VR headset apredetermined set of VR keys for user interaction. Alternatively, theuser may be prompted by a display of keys and commands on the displayassociated with the second embodiment to type the corresponding keys onthe physical keyboard. The predetermined set may include at least onepass through every available VR key and command, including keycombinations such as CTRL-ALT-DEL on the VR keyboard. As the user entersevery key combination on the physical keyboard while wearing theactivated VR glove, the VR keyboard system and method get VR glovepositions corresponding to the physical keys being depressed by theuser.

[0117] The physical keyboard is provided with appropriate powerconnections such that, as the physical keys are being sequentiallydepressed, the physical keyboard generates corresponding controlsignals, which are sampled by the VR keyboard system and method inresponse to the depression of the keys prompted by the sequentialdisplay of VR keys to the user through the VR headset. Once the controlsignals are sampled, the VR keyboard system and method store the controlsignals in a memory, and generate a mapping of the VR glove positionswith the VR keys, associated key codes, and the control signals.

[0118]FIG. 33 illustrates a flowchart of a method using sampled controlsignals to operating the VR keyboard, in which a VR glove position isreceived, and a corresponding key code is determined therefrom using themapping stored in the memory. The corresponding control signal isretrieved from the memory, and applied as a control signal or controlcode to the application program; that is, the control signal is sent tothe application program and/or the second processor as though it isgenerated and transmitted by an actual keyboard, when in fact it isgenerated by the VR-keyboard system and method. The VR keyboard systemand method also generate a VR display indicating actuation of a VR keycorresponding to the key code.

[0119]FIG. 34 illustrates a mapping of glove positions with displayedkeys and sampled control signals using the method of FIGS. 32-33, inwhich a displayed key provided to the user is associated with a key codegenerated by the VR keyboard system and method, and associated with thecontrol signals sampled from the actual keyboard. The key codes may bearbitrary, or may be ASCII values, with the key codes serving as anindex of the displayed key with the glove positions and sampled controlsignals. For example, a capital “A” may be assigned a key code “1”, and,after detection of the corresponding VR glove positions, is associatedwith a glove position (X1, Y1, Z1) corresponding to the user moving theVR glove to such a position in space. Concurrently, the VR gloveposition, the key code, and the displayed key are associated in themapping with a control signal sampled from the physical keyboard. Thesampling may be performed by a digital sample-and-hold circuit. Forexample, the control signal associated with the capital “A” may be adigital bit sequence such as “10010001”. Using such sampled controlssignals, the disclosed VR keyboard system and method may be adapted forVR keyboard data and command input for any computing system whichheretofore uses a physical keyboard, since the VR keyboard system andmethod generates substantially identical control signals for input tothe computer system.

[0120]FIG. 35 illustrates a seventh embodiment of the VR keyboard systemand method with a VR mouse. FIG. 36 illustrates the seventh embodimentusing the VR glove to operate the VR mouse, in which a predeterminedglove configuration such as a clenched fist, upward thumb, and extendedindex finger in the VR glove corresponds to operation of the VR mouse.Otherwise, the VR glove positions correspond to VR keyboard inputs.Alternatively, the disclosed VR keyboard system and method may detectglove positions and distinguish hand and glove orientations for keyboarduse from hand and glove orientations for mouse use. FIG. 37 illustratesVR glove positions for keyboard use, in which the palm of the VR gloveis substantially planar, with the fingers extending downward, relativeto the palm. FIG. 38 illustrates VR glove positions for mouse use, inwhich the pal of the VR glove is substantially curved with the fingersextending about and/or surrounding at least a portion of the mouse.

[0121]FIG. 39 illustrates a flowchart of a method for operating theseventh embodiment using either a VR keyboard or a VR mouse. The methodgets the VR glove position, and determines if the VR glove positioncorresponds to a VR keyboard orientation or VR mouse orientation. If aVR mouse is to be used, then the method performs a VR mouse subroutineto receive VR glove positions as VR mouse movements and VR mouse buttonclicks and activations to generate corresponding mouse signals.

[0122] The VR mouse signals are applied to the second processor and/orthe application program, for example, through mouse driver software, andthe method then displays a VR mouse display which is changed and/ormoved in the VR world by being displayed through the LCD of the VRheadset to reflect the corresponding mouse commands, such as movement ofa VR cursor or VR arrow on the LCD of the VR headset. The method maythen loop back to process more VR glove positions.

[0123] However, if VR keyboard use is detected, the method performs theVR keyboard subroutines, described herein, to generate VR keyboardsignals which are applied to the processor, with the VR keyboard beingchanged or updated to display the corresponding VR keyboard commands andVR key actuations.

[0124] Accordingly, since the VR keyboard and/or the VR mouse maygenerate keyboard and mouse signals, respectively, which may be input tokeyboard driver software and mouse driver software, respectively, suchkeyboard and/or mouse signals may be input to the application programthrough any port with the respective software drivers configured toreceive and process such keyboard and/or mouse signals therefrom,respectively. For example, the first processor may be connected to thesecond processor through a serial port, a COM port, a modem port ortelephone jack, etc., and the appropriate driver software may beconfigured to receive data signals from such ports or jacks.Accordingly, a computer implementing the disclosed VR keyboard systemand method may not require a keyboard port and/or a mouse port.Furthermore, such keyboard ports and mouse ports may be eliminated, thussaving on hardware implementations for such hardware-based actualkeyboards and actual mouses/mice. In addition, the disclosed VR keyboardsystem and method may be implemented using “PLUG-AND-PLAY” technology,with the second processor capable of recognizing a connection to thedisclosed VR keyboard system through any available port, and so thesecond processor may be automatically or manually reconfigurable tooperate with the disclosed VR keyboard system in many diverse hardwareconfigurations, include configurations without keyboard and/or mouseports.

[0125]FIG. 40 illustrates an eighth embodiment of a VR keyboard and VRmouse for use with an actual keyboard and actual mouse, in which anactual keyboard and/or mouse are connected to a logic circuit, which isalso connected to the first processor. The logic circuit receiveskeyboard signals from the actual keyboard and, through the firstprocessor, from the VR keyboard. The logic circuit receives mousesignals from the actual mouse and, through the first processor, from theVR mouse. The logic circuit may be an exclusive-OR (XOR) gate, or otherlogic gate circuits or switches, for switching or gating keyboardsignals and/or mouse signals to the second processor.

[0126]FIG. 41 illustrates a flowchart of operation of the eighthembodiment with a method to respond to either the VR keyboard or theactual keyboard, in which VR keyboard signals and actual keyboardssignals are received, and then applied to the logic gate or circuit togenerate a single key input signal to the second processor. Similarly,FIG. 42 illustrates a flowchart of a method to respond to either the VRmouse or the actual mouse, in which VR mouse signals and actual mousesignals are received, and then applied to the logic gate or circuit togenerate a single mouse input signal to the second processor.

[0127] For example, using an XOR gate, signals from either the VRkeyboard or the actual keyboard but not both are gated to the secondprocessor. Similarly, using an XOR gate, signals from either the VRmouse or the actual mouse but not both are gated to the secondprocessor. In this manner, if the user operates both VR devices andactual devices, the XOR gate of the logic-circuit prevents conflictingsignals from going to the second processor. If a conflict occurs, nosignal is sent to the second processor.

[0128] Alternatively, the logic circuit may implement logic IF-THEN orlogic IF-THEN-ELSE statements, such that if there is a VR keyboardsignal being input to the second processor, then the actual keyboardsignal is ignored, else the second processor receives and uses theactual keyboard signal. Accordingly, VR keyboard signals may be givenprecedence if present. Such logic IF-THEN or IF-THEN-ELSE statements mayalso be used conversely for ignoring the VR keyboard signals and givingkeyboard signals from an actual keyboard presence if present. SuchIF-THEN operations may also be used with signals from an actual mouseand a VR mouse, to give precedence or to resolve signal conflicts.

[0129]FIG. 43 illustrates a ninth embodiment of the disclosed VRkeyboard system and method for displaying VR hand images using the VRkeyboard, in which the first processor uses VR hand image data, whichmay be stored in a memory of the first processor. FIG. 44 illustrates aflowchart of the method of operation of the ninth embodiment displayinga VR hand image, in which the method gets a VR glove position, generatesa corresponding key code from the VR glove position using apredetermined mapping, and sends the key code to the applicationprogram. The VR processor then generates a corresponding VR hand imagefrom the VR glove position, with the VR hand image being, for example, asmooth, idealized, and generic image of a hand. The method thengenerates a display representing the key input in VR, including the stepof generating the key input display on a VR representation of a keyboardin the VR headset with the VR hand image representing the VR actuationof the corresponding VR key.

[0130]FIG. 45 illustrates a flowchart of a method for generating the VRhand images of the ninth embodiment using, for example, actual hand datafrom the user. For example, the hands of the user may be pre-scanned andstored in the memory of the disclosed VR keyboard system and method.During use, the pre-scanned hand image of the actual hands of the useris retrieved from the memory, and using morphing techniques, thepre-scanned hand image is morphed to correspond to the VR glove positionwith corresponding VR fingers extended to actuate a VR key on the VRkeyboard. Accordingly, the specific user may experience, by visual cuesand views, the actual operation a keyboard with his/her own hands, whenin fact the user is operating a VR keyboard. In conjunction with forcefeedback implementations of the VR gloves, the user may also experienceand feel operation of an actual keyboard, when in fact the user isseeing a VR image of his/her own hands, and is feeling a VR actuatedtactile response, through the actuators, to pressing VR keys.

[0131]FIG. 46 illustrates a tenth embodiment of the disclosed VRkeyboard system and method using sensors to detect hand positions of auser without a VR glove. The sensors may include magnets,accelerometers, or other mechanisms for detecting the relative movementof positions of the hand of the user.

[0132]FIG. 47 illustrates an alternative of the tenth embodiment using acamera and machine vision to detect hand positions of a user without aVR glove, with the first processor performing image processingtechniques to translate, the images from the camera into parametersdetermining the positions and orientations of the hands and portionsthereof. The camera may be an optical/visible light camera, an infraredcamera, and/or a camera using any electromagnetic wavelength and/orfrequency to generate an image of the hand. Alternatively, the hands ofthe user may be coated with, preferably, relatively safe chemicals whichare readily detectable by a corresponding type of camera. The user maybe instructed, for example, by automated instructions, to dabfluorescent chemicals on the tips and knuckles of each finger fordetection by a fluorescent-light sensitive camera.

[0133]FIG. 48 illustrates another alternative of the tenth embodimentusing an infrared detector to detect hand positions of a user without aVR glove, in which the infrared energy/body heat generated by the handsof the user may be detected and translated into relative positions andorientations of the hand. Other sensor systems may be used, such asDoppler radar, sonar, Doppler sonar and ultrasound to detect the handsof the user.

[0134]FIG. 49 illustrates a schematic of the tenth embodiment using handsensors for operating a VR keyboard, in which hand sensor, such as thesensors, cameras, and detectors of FIGS. 46-48, detect the hands of theuser. The first processor includes a hand sensor data processor forgenerating hand positions and orientations, such as (X, Y, Z)coordinates, corresponding to key inputs, as described herein withrespect to the mappings, embodiments and implementations using the VRgloves.

[0135]FIG. 50 illustrates a flowchart of the operation of the tenthembodiment for scanning hand positions of the hands of user to operatethe VR keyboard. The method scans the hand positions of the user, forexample, using the hand sensors applied to the hands of the user as inFIG. 46. The met-hod then generates hand position data from the handpositions, for example, in the form of position and orientation vectors(X, Y, Z). The method then determines a key code corresponding to thehand position data using a predetermined mapping. The key code is thensent to the application program and/or second processor, and the methodgenerates the display representing the key input, such as a VR key inputon a VR representation in the LCD of the VR headset.

[0136] While the disclosed VR keyboard system and method is particularlyshown and described herein with reference to the preferred embodiments,it is to be understood that various modifications in form and detail maybe made without departing from the scope and spirit of the presentinvention, such that the present invention encompasses any and allmachines, articles of manufacture, compositions of matter, processes,apparatus, systems, devices, computers, structures, components, meansmethods, algorithms, techniques, steps, routines, hardware, firmware,software, computer programs, objects in object oriented programming,network implementations, distributed computing implementations, Internetapplications, multimedia applications, heterogeneous configurations andsystems, terrestrial and extraterrestrial applications, wire-basedimplementations, wireless-based implementations, fiber-optic-basedimplementations, nanotechnology-based implementations, cyberneticimplementations including human-machine interfaces and interconnections,combinations and hybrid implementations of any and all of theimplementations suggested herein, etc., including anything under the suncapable of generating, operating, and/or providing a virtual realitykeyboard. Accordingly, modifications such as any examples suggestedherein, but not limited thereto, are to be considered within the scopeof the present invention.

What is claimed is:
 1. A system for implementing a virtual reality (VR)keyboard, the system comprising: a VR headset including at least onedisplay and worn by a user with the at least one display viewable by theuser, with the VR headset responsive to first image data, for displayingthe VR keyboard to the user through the at least one display, with theVR keyboard having a first appearance corresponding to the first imagedata; a VR input device, responsive to motion of a portion of the bodyof the user, for generating input signals corresponding to the motion;and a processor connected to the VR headset and operating a VR keyboardgenerating program for providing the first image data to the headset,the processor being responsive to the input signals from the VR inputdevice, for generating motion image data corresponding to the motion;and wherein the VR headset is responsive to the motion image data forgenerating the VR keyboard having a second appearance corresponding tothe motion of the portion of the body of the user.
 2. The system ofclaim 1, wherein the VR input device is responsive to the motion of thefingers of the user corresponding to keystrokes to generate the inputsignals; wherein the processor, responsive to thekeystroke-corresponding input signals, generates finger-motion imagedata; and wherein the VR headset is responsive to the finger-motionimage data for generating the VR keyboard having the second appearancerepresenting depressed VR keys on the VR keyboard, thereby displayingdepressed VR keys in response to VR keystrokes.
 3. The system of claim2, wherein the processor, responsive to the input signals from the VRinput device, generates corresponding input data output to externaldevices with the input data corresponding to the VR keystrokes.
 4. Thesystem of claim 3, wherein the input data corresponds to at least one ofa command, text, and a graphic user interface signal.
 5. The system ofclaim 3, wherein the external device is a graphic user interfaceresponsive to the input data corresponding to mouse operations and text.6. The system of claim 1, wherein the processor generates VR world data;and wherein the VR headset generates a VR world corresponding to the VRworld data, with the VR keyboard displayed in the VR world.
 7. Thesystem of claim 1, wherein the processor, responsive to a selectedmapping chosen from a plurality of predetermined mappings, generates thefirst image data corresponding to the VR keyboard having VR keyscorresponding to the selected mapping; and wherein the VR headset,responsive to the first image data, displays the VR keyboard having VRkeys corresponding to the selected mapping.
 8. The system of claim 7,wherein the predetermined mappings include specifications for VR keyscorresponding to different language.
 9. The system of claim 7, whereinthe predetermined mappings include specifications for VR keyscorresponding to different predefined computer commands.
 10. The systemof claim 7, wherein the predetermined mappings include specificationsfor positioning a plurality of keys having different positions on acommon keyboard shape.
 11. The system of claim 7, wherein thepredetermined mappings include specifications for defining different VRkeyboard shapes and key orientations displayed in virtual reality. 12.The system of claim 2, wherein the VR input device includesforce-feedback means for selectively generating pressure to the fingersof the user during motion of the fingers; wherein the processor,responsive to the input data from the VR input device, controls theforce-feedback means.
 13. The system of claim 2, wherein the processorincludes: a neural network, responsive to the input signals, forlearning a custom three-dimensional orientation of at least one hand andat least one finger of a predetermined user during use of the VRkeyboard by the predetermined user; and wherein the processor,responsive to the input signals applied to the trained neural network,for predicting the VR keys to be depressed corresponding to the motionof the keystroke-corresponding input signals, and for generating thecorresponding finger-motion image data.
 14. The system of claim 13,wherein the trained neural network, responsive to the input signals,authenticates a current user as the predetermined user.
 15. The systemof claim 1, wherein the VR headset includes: an orientation sensor,responsive to a vertical orientation of the head of the user wearing theVR headset, for generating an orientation signal; and wherein theprocessor, responsive to the orientation signal, for generating thefirst and second image data in response to the vertical orientationbeing within a predetermined vertical range, and for not generating anyof the first and second image data in response to the verticalorientation being outside of the predetermined vertical range.
 16. Thesystem of claim 1, further comprising: a switch for switching inputsignals to the processor either from the VR input device or from aphysical keyboard.
 17. The system of claim 2, wherein the processorgenerates VR hand image data corresponding to the motion of the fingers;and wherein the VR headset, responsive to the VR hand image data,generates VR images of at least one VR hand having at least one VRfinger positioned substantially adjacent to the VR keyboard, with the atleast one VR finger appearing to depress the depressed VR keyscorresponding to the motion of the physical hand of the user.
 18. Asystem for implementing an interactive virtual reality (VR) keyboard,the system comprising: a VR headset including at least one liquidcrystal display (LCD) and worn by a user with the at least one LCDviewable by the user, with the VR headset responsive to first imagedata, for displaying the VR keyboard to the user through the at leastone LCD, with the VR keyboard having a first appearance corresponding tothe first image data; a VR input device, including a VR glove,responsive to motion of the fingers and hands of the user correspondingto keystrokes, for generating input signals corresponding to the motion;a processor connected to the VR headset and operating a VR keyboardgenerating program for providing the first image data to the VR headset,the processor being responsive to the keystroke-corresponding inputsignals, generates finger-motion image data; and wherein the VR headsetis responsive to the finger-motion image data for generating the VRkeyboard having the second appearance representing depressed VR keys onthe VR keyboard, thereby displaying depressed VR keys in response to VRkeystrokes.
 19. A method for providing an interactive virtual reality(VR) keyboard, the method comprising the steps of: operating a VRkeyboard generating program using a processor; outputting first imagedata to a VR headset having a display; displaying the VR keyboard on thedisplay to the user, with the VR keyboard having a first appearancecorresponding to the first image data; detecting motion at a VR inputdevice, with the motion due to movement of the fingers and hands of theuser corresponding to keystrokes; generating input signals at the VRinput device corresponding to the motion; generating finger-motion imagedata using the processor corresponding to the input signals; andgenerating the VR keyboard on the display of the VR headset using thefinger motion data, with the VR keyboard having a second appearancerepresenting depressed VR keys on the VR keyboard, thereby displayingdepressed VR keys in response to VR keystrokes.
 20. The method of claim19, wherein the step of displaying the VR keyboard on the display to theuser includes the step of: displaying a selected keyboard image from aplurality of keyboard images.